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ABSTRACT 

This paper uses a multi-level model to identify 
course variables that affect the relationship between study 
activities and achievement. Data on course characteristics were 
collected from 12 high school biology classes, and study activity and 
achievement measures were collected from 136 students enrolled in 
these classes. The results indicate that more extensive feedback on 
quizzes was associated with increases in the relationship between 
effort management study activities and achievement. However, feedback 
on homework was linked to decreases in the effectiveness of both 
effort management and autonomous management study strategies. The 
provision of extra time for student questions during teacher-led 
reviews increased the effectiveness of autonomous management 
strategies. Finally, increases in the number of content categories on 
the teachers' test led to a decrease in the relationship between 
memory augmentation strategies and achievement. The findings suggest 
that a multi-level analysis provides a more comprehensive and 
appropr i at e explanat i on of the hierarchical relati onship between 
course characteristics, study activities, and achievement. (Contains 
9 figures and 23 references.) (Author) 
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Abstract 

This paper uses a multi-level model to identify course variables that affect the 
relationship between study activities and achievement. Data on course 
characteristics was collected from 12 high school biology classes, and study activity 
and achievement measures were collected from 1 36 students enrolled in these 
classes. The results indicate that more extensive feedback on quizzes was 
associated with increases in the relationship between effort management study 
activities and achievement. However, feedback on homework with linked to 
decreases in the effectiveness of both effort management and autonomous 
management study strategies. The provision of extra time for student questions 
during teacher-led reviews increased the effectiveness of autonomous management 
strategies. Finally, increases in the number of content categories on the teachers' 
test led to a decrease in the relationship between memory augmentation strategies 
and achievement. The findings suggest that a multi-level analysis provides a more 
comprehensive and appropriate explanation of the hierarchical relationship between 
course characteristics, study activities, and achievement. 



Course characteristics, study activities, achievement 

3 

Explaining the Relationship between Students' Study Activities, Achievement, and 
Course Characteristics Using a Multi-Level Model 
A question of obvious import for both educational theory and practice is 
how course variables affect student study activities and in turn, how study 
strategies impact academic achievement. However, previous research on the 
relationship between course characteristics, study activities, and achievement has 
searched for direct links between only two of the three variables in isolation. The 
traditional approaches to investigating the relationships between these variables 
separately is portrayed in the upper part of Figure 1 . More specifically, researchers 
have studied (1) the relationship between instructional features and study activities 
(e.g., Entwistle & Ramsden, 1983; Hamaker, 1986; Martin & Saljo, 1976; 
Thomas, Bol, Warkentin, Wilson, Strage, & Rohwer, in press), (2) the 
relationship between instructional features and achievement (e.g., Mayer, 1979; 
Rothkopf, 1973; Thomas et al., in press), and (3) the relationship between study 
activities and achievement (Crooks, 1988; Rohwer & Thomas, 1989; Thomas et 
al., in press). In most cases, regression or ANOVA has been employed to 
investigate these relationships. In this paper we consider the possibility of a 
somewhat different relationship among course characteristics, study activities, and 
achievement. Our conception is that students develop a style of study in response 
to particular course characteristics which affects the relationship between students' 
study style and their achievement. Specifically, we ask whether course 
characteristics might affect the way students' study activities are related to 
achievement. That is, we ask: Is the relationshio between study activities and 
achievement systematically associated with differences in course characteristics? 
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This multi-level relationship among the three types of variables is also illustrated in 
Figure 1. 



Insert Figure 1 about here 



This kind of question is more appropriately addressed by using a multi-level 
approach which models two distinct levels of influence on student achievement 
(Bryk&Raudenbush, 1992; Raudenbush, 1988). We conceptualize the way the 
multi-level analysis works as follows. First, the multi-level model estimates the 
within-class relationship between selected student study variables and student 
achievement. At a second level, the multi-level model estimates the between-class 
effects by regressing these study activity-achievement relationships on selected 
course variables identified in classrooms. (The actual estimation algorithm 
addresses these two levels simultaneously). This approach not only answers a 
different question to that of traditional analyses, it models the naturally hierarchical 
nature of data collected on the effectiveness of classroom practices, in this case, 
student performance measured at the individual level and characteristics of courses 
assessed at a class or group level. 

The employment of multi-level techniques to analyze hierarchical data has 
become increasingly popular with the realization that both between and within level 
effects should be considered simultaneously in order to provide a more 
comprehensive and appropriate explanation of the inherently hierarchical 
relationship among variables. For example, multi-level models have been used to 
investigate the relationship between student characteristics and student outcomes as 
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a function of school level variables (Bryk & Raudenbush, 1989; Bryk & Thum, 
1989;Raudenbush&Bryk, 1986, 1988). These researchers (Bryk & 
Raudenbush, 1 987, 1989) have also proposed and demonstrated that multi-level 
techniques allow one to assess psychological change and how it is linked to 
background characteristics. Similarly, Burstein ( 1980) reviews findings in school 
effects research, large scale program evaluation, and instructional effects research 
and argues that a multi-level approach is beneficial for specifying more appropriate 
models for multi- level data. Thus, multi- level modeling represents a 
methodological advance in understanding multi-level relationships in various areas 
of educational and psychological research. 

Though multi-level modeling has been employed to study various classes of 
within and between level variables, it has never been utilized to examine how 
course variables influence the relationship between student study activities and 
achievement. Therefore, the purpose of the present study is twofold: ( 1) to explore 
the kinds of course variables that either promote or impede productive study 
strategies, and (2) to illustrate how multi-level modeling can be used to explain the 
relationship between student study activities and achievement as moderated by 
course characteristics. 

In our research we have identified three categories of course characteristics 
which are hypothesized to influence study activities and achievement (Strage, Tyler, 
Thomas & Rohwer, 1987;Thomas, Bol & Warkentin, 1991;Thomas et al., in 
press). The first category is composed of the demands imposed on students to meet 
course requirements and standards. Examples of course demands are the amount of 
material to be leaded, the degree of cognitive challenge associated with teachers* 
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tests, and the salience of different types of knowledge products on these tests (e.g., 
knowledge of facts versus knowledge of principles). The second category of 
course characteristics is composed of su pports . Supports consist of teacher 
provisions such as practice opportunities and feedback that prompt and maintain 
student engagement in study activities appropriate to the demands of the course. 
The third category of course characteristics we refer to as compensations . 
Compensations are defined as provisions that act to abrogate the need for students 
to engage in demand- responsive study activities. For example, allowing students to 
refer to their text or notes while taking an exam (open-book tests) might dissuade 
students from engaging in memory augmentation activities while preparing for that 
test. 

We predict that the effectiveness of engaging in different study activities will 
vary as a function of course characteristics found in high school biology 
classrooms. The presence of certain Characteristics, such as high demands and high 
supports, should enhance the relationship between study activities and achievement, 
whereas the presence of other characteristics, such as the presence of compensatory 
practices, may diminish this relationship. 

Method 
Participants 

Participants were 1 2 teachers and 1 36 of their students from seven high 
schools in the San Francisco Bay Area. Each teacher taught one or more sections 
of a general biology course. A sample of iO-25 students were randomly selected 
from each of the 1 1 courses to serve as the student sample. The grade level of 
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students enrolled in these courses ranged from Grade 9 to Grade 12, with the 
majority of students coming from the 10th grade. 

Instruments and Data Sources 

Students' Study Activities 

Students' study activities were measured using three of six scales that make 
up the Study Activity Questionnaire (SAQ): (1) effort management, (2) memory 
augmentation, and (3) autonomous management. These scales are described briefly 
below. Additional information on the SAQ and its scales can be found in Thomas 
et al. (in press). 

Effort Man agement . This is a hierarchical dimension relating to the tendency 
of students to match their study habits to the demands of the situation. The 
dimension has three components and four levels. The three components are time, 
concentration, and learning effectiveness. The four levels are (a) self-monitoring- 
the disposition to pay attention to time, concentration, or relative mastery; (b) self- 
regulation-the disposition to take steps to correct difficulties in time allocation, 
concentration, or perceived mastery; (c) planning- the disposition to take steps to 
adopt a strategy in advance of studying to deal with possible time, concentration, or 
learning effectiveness difficulties; and (d) evaluation- the disposition to engage, 
after studying, in assessment of the relative success of practices instituted to 
manage time, concentration, or learning effectiveness. 

Memory Augmentation . This is a study activity dimension indexing the 
disposition to engage in activities to make to-be-learned material more memorable. 
The dimension ranges from (a) no engagement, (b) duplicative activities (e.g., 
repeating material over and over), (c) interpretive activities (putting material in one's 
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own words), to (d) constructive activities (e.g., making up study aids). A high 
score for memory augmentation reflects the studeni. ' disposition to engage in 
generative activities intended to aid memory. 

Autonomo us Management . This is a dimension measuring a student's 
disposition to use study time to engage in self-directed activity (e.g., preparing 
study material, testing oneself) as opposed to doing assigned activity (e.g., reading, 
doing homework). A high score in autonomous management reflects the 
disposition to take control over one's learning. 
Course characteristics . 

Three course characteristics are described briefly below; Note that we 
actually estimated HLM effects for additional course characteristics hypothesized to 
influence studying and achievement. Only those that proved to be stable effects are 
listed and described here. A description of the full set of course features 
hypothesized to affect the study-achievement relationship is provided in Thomas et 
al. (in press). 

Feedback. This support variable is a summary measure that represents that 
extent of feedback that teachers provided to students on their coursework. In 
interviews, teachers were asked to describe the type of feedback provided to 
students on quizzes, homework, and tests. The alternatives provided for this 
questions were: no feedback (scored as 0), an overall score or grade (scored as 1), 
a score for each item (scored as 2), and the provision of written comments (scored 
as 3). This scoring scheme also yielded a measure of the extent of feedback 
provided separately for quizzes, homework, and tests. 
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Extra time provided for student questions during review . This support 
variable measures whether teachers provided extra time for students to ask 
questions during the teacher-led review session immediately prior to the genetics 
unit test in the course. Data on this dichotomous variable (yes/ho) was gathered 
during the teacher interview on the genetics unit. These responses were cross- 
referenced with our observation of the review session. 

Number o f content categories on the test . This variable was classified as a 
demand characteristic that referred to the number of discrete biology topics covered 
on the teacher-administered unit test covering the genetics unit. Scores on this 
measure were obtained by a content analysis of teachers* tests on the genetics unit. 
Student Achievement 

Achievement was measured using a researcher-developed test of 
achievement in genetics. The Genetics Achievement Test (GAT) was developed in 
order to have a valid way of comparing students' knowledge and capabilities across 
the 12 participating courses and to assess the degree to which qualitative differences 
in study activities resulted in differences in achievement capabilities. Accordingly, 
the content covered by the test represented content that received similar weight and 
treatment across courses and textbooks. The nine items that made up the test were 
constructed to represent three cognitive processing (encoding/selection, integration, 
extension) and three representational (facts/details, concepts^definitions, principles) 
levels assessed with the SAQ. 

Procedure 

The investigation was conducted during the administration of one particular 
unit in each course, the genetics unit. Students' study activities were assessed eariy 
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in the academic year, prior to the start of the genetics unit, and then again at the 
completion of the unit. Data relevant to course characteristics were collected during 
and at the end of the instructional unit. These data were collected by means of 
classroom observation of the unit test review session, document analysis of all 
materials related to the unit, and teacher interviews focusing on classroom practices 
during the genetics unit. The Genetics Achievement Test was also administered at 
the end of the unit. 

Method of Analysis 
According to our conception, the effectiveness of engaging in various study 
activities may vary across courses depending on the pattern of demands, supports, 
and compensations that make up these courses. Note that predictions of this kind 
pertain to influences on the criterion performance of individuals that stem from two 
distinct levels, an individual differences level (study activities) and a course level 
(course characteristics). Such multi-level data can be appropriately analyzed using 
methods based on a hierarchical linear model (HLM) analysis. Suppose that we 
represent the performance of student i in classroom j on the criterion variable by Yjj 
(we follow the notation of Bryk & Raudenbush, 1992). Then if Xy is that 
student's measure on a selected SAQ variable, we use a within-class equation to 
model dependence of achievement on study activity: 

Yij = Pjo + PjiXij + rij (l) 
where rjj is a random error associated with student i in class j. Note that this can be 
done within each classroom, so we end up with as many intercepts (Pj 0 ) and linear 
regression coefficients (pj[) as there are classrooms. A selection of these 
regressions for a particular study skills variable (effort management) is shown in 



Course characteristics, study activities, achievement 

11 



Figure 2. The^e regressions have been arranged down the page in a particular way: 
At the top is a classroom where the teacher provides written feedback on quizzes. 
In the next two classes, the teacher provides a score for each item in the quiz. In the 
next, the teacher provides only an overall score on the quiz, and, finally, in the 
bottom one, the teacher provides no feedback at all. What is interesting about the 
regressions that are estimated within each of these classes is that the regression 
slope becomes more positive (i.e., steeper) as the quality of feedback increases. 
This is exactly the sort of systematic variation between two variables (in our case 
study activities and achievement) as influenced by a third variable (in our case 
course characteristics) that is appropriate for multi-level analysis. 



Insert Figure 2 about here 



We can display this in a more compact form by plotting the regression 
slopes as a function of the course characteristics variable, feedback on courses. 
Figure 3 shows where each of the five within-class regressions from Figure 2 occur 
on this plot, and Figure 4 gives all of the within-class regressions, with each one 
represented merely as a dot. Thus, we can interpret this to imply that the effect of 
effort management on student performance is enhanced by higher quality teacher 
feedback on quizzes. This analysis occurs at the between-class level, and can be 
represented by the following equation: 

Pjk = Yok + YlkWj+u jk (2) 
where y 0 and yik are the between classroom intercept and linear regression 
coefficient for within-class effect k, respectively (in our case k is either 0 or 1 for 
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within-class intercept and slope, respectively, and we shall concentrate on the 
slope), Wj is the measure on the course characteristic for class j, and u jk is a 
random error associated with class j and within-class regression weight Pjk . Thus, 
when we noted in Figures 2-4 that there was a positive relationship between the 
within-class regression slope and quality of feedback, this corresponds to noting 
that y Ik is estimated to be positive. This procedure could be done step-wise with 
many regressions at the within-class level and one at the between-class level. The 
HLM approach uses an iterative procedure that effectively estimates both levels 
simultaneously. This can be shown to be more appropriate where the data conform 
to the error assumptions in equations ( 1 ) and (2) (Bryk & Raudenbush, 1 992). 
Moreover, the HLM approach provides a statistical framework in which one can 
test whether the observed relationship is statistically significant (Bryk & 
Raudenbush, 1992). 



Insert Figures 3 and 4 about here 



The present data were analyzed using the computer program, HLM2 (Bryk 
Raudenbush, Seltzer & Congdon, 1 986). In an attempt to control for pre-existing ' 
differences in the naturally occurring classes, and in order to isolate potential effects 
of the course characteristics for the particular unit under study, we use as within- 
class predictor, (Xys above) differences in student study variables. Thus, we are 
modeling the effect of changes in students' study activities on students- 
performance (Yys above) at the individual level. According to the teachers in the 
sample, this unit is the first time that students have encountered genetics in their 
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school curriculum. Thus, we assume for interpretative purposes that all students 
taking a pre-test using the researcher-designed achievement test would receive a 
score of zero. Scores on the GAT were analyzed by the HLM2 program, with the 
each of the separate SAQ scale scores used as the within-level predictor variable. 
Thus, for a given SAQ scale and course characteristic, all other things being equal, 
an estimate of a statistically significant y )k , indicates that, assuming the estimate is 
positive, changes in study behavior (as measured by the SAQ scale) become more 
effective (i.e., result in greater increases in achievement per unit increase in SAQ 
scale) with increases in the course characteristic variable. Note that this 
interpretation relates only to the rate of increase, yi k- Interpretation may be 
somewhat more complicated when the intercept , y 0k , is also found to be different 
from zero. In the analyses reported below, yok was found to be non-significant 
except where reported otherwise. 

Results 

The course characteristics (the between-level variables) used in these 
analyses were selected as those hypothesized to affect student engagement in the 
different levels and dimensions of the study activity hierarchies. Use of every 
course characteristic with each SAQ subscale would have proved prohibitive in 
cost, so the course characteristic framework was used as a guide for the selection of 
course variables that were expected to be relevant for each SAQ scale. Though a 
large number of analyses were completed, only those significant at the 5 percent 
level are reported in this paper. These should not be considered strict statistical 
significance tests, as there was no proper sampling design for either level, and as 
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the analyses share a great number of variables. Rather, the analyses should be 

considered to be exploratory. 

There were five statistically significant course characteristic regression 

coefficients at the 5 percent significance level. The summary of results that follows 

is organized by the pairs composed of the study activity and course characteristic 

variables from each analysis. 

Effort Management Change Score s on the SAO; Feedback on O niyyes 
A higher level of feedback on quizzes is associated with increases in the 

relationship between effort change scores and achievement (see Figure 4). The 

more feedback provided on quizzes the more effective are effort management scores 

for enhancing achievement. 

The provision of feedback to students on their coursework has been 
acknowledged as an effective teaching practice for enhancing performance (Crooks, 

1988; Kulhavy, 1 977). There is also some evidence suggesting that feedback is 
associated with more productive types of study behaviors. Duckworth, Fielding, & 
Shaughnessey (1986) found that the students' ratings of teacher feedback were 
positively correlated with their ratings of effort expended in the course and with 
their perceptions of efficacy in the course. The present finding extends these 
results, suggesting that extensive feedback practices prompt students to engage in 
diligent effort management activities which, in turn, is associated with high 
achievement. 

Effort Management Change. s™ r ^ on the SAO: Feedback on Homework 
Higher levels of feedback on homework are associated with decreases in the 
relationship between Effort Management change scores and achievement (see 
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Figure 5). More feedback provided on homework is associated with diminishing 
effectiveness of changes in Effort Management scores for enhancing achievement. 

Ostensibly, this result contradicts the hypothesis that high levels of support 
provided by teachers on students' coursework, including homework, would be 
associated with more effective change in reported Effort Management activities. 
However, closer examination of the results shows that this apparent contradiction 
can be resolved. Note that Figure 5 shows the relationship between Feedback on 
Homework and the regression coefficients relating change in Effort Management to 
achievement (i.e.., y , k in (2)), but this is not the only coefficient in (2): We must 
not forget the intercept Yok- In this case, the intercept is also significantly different 
from zero, indicating that interpretation must include the effect of this coefficient. 
Figure 6 shows the fitted linear regressions relating change in Effort Management to 
achievement for each of the values for Feedback on Homework. Examination of 
the figure shows that the "effect" of giving differing levels of feedback is, indeed, 
in line with our prediction (that is higher levels of feedback are associated with 
higher achievement for a given level of change in Effort Management), except at the 
highest levels of change. But, it is also true that the effectiveness of changes in 
Effort Management diminishes with increasing feedback at a value on the change in 
Effort Management scale of approximately 2. This diminishment results in a 
reversal of the "effect" of feedback, so that beyond this point, giving higher levels 
of feedback is counterproductive. This point on the change in Effort Management 
scale is one that few students in our sample attained»a typical student at this point 
would have had to give the most extreme possible responses (negative on pretest, 
positive on post-test) to all, or all but one, of the questions on the scale. Thus, we 
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can say that for most students, giving higher levels of feedback on homework is 
associated with higher achievement at a given level of change in Effort 
Management. However, when the student makes very dramatic changes in Effort 
Management, ihe relationship is reversed. 

It should be noted that in the Duckworth et al. ( 1 986) study, the student 
ratings pertained to feedback provided on tests, no. on homework or other 
coursework. I, may be the case tha, students more frequently attend ,o and modify 
their study activities based on feedback received on tests or quizzes rather than on 
homework, which is a more routine class activity. Another possibility is tha, the 
type of feedback (e.g. , the content and extent of written comments) provided to 
students may systematically differ for homework versus quizzes and tests. In fact, 
if teacher tend to give high quality feedback on either quizzes or homework, bu, 
no. both (which is indeed ,he .rend in our small sample), then this wou!d explain 
the apparently contradictory relationship between this result and the previous one. 



Insert Figures 5 and 6 about here 



The provision of more extensive feedback on homework is associated with 
decreases in the relationship between change scores on ,he Autonomous 
Ma^gemen, scale and S ,ude„, perfonnance on ,he achievemen, measure ( S ee Figure 
7). These findings are similar to those reported for the effect of homework 
feedback on the relationship between change, in Effort Management scores and 
achievement. In fad, ,he situation is almost identical, with a significant intercept 
giving a rela,io„ sh ip ,ha, is very similar ,o tha, shown in Figure 6, except tha, the 
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point at which all the regressions intercept is somewhat further to the right, at ; 
change-score where no student actually scored on the Autonomous Manager 
scale. Thus, the interpretation should be the same as that for Effort management. 
The "effect" of giving differing levels of feedback is in line with our prediction (that 
is, higher levels of feedback are associated with higher achievement for a given 
level of change in Autonomous Management), except at the highest levels of 
change. But, it is also true that the effectiveness of changes in Autonomous 
Management diminishes with increasing feedback. These results are complicated 
and may require further research on the types of feedback provided on different 
course assignments. Perhaps the provision of extra homework feedback tends , 
act as a ceiling on the possible effects of increases in Autonomous Management 



to 
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activities. 



Insert Figure 7 about here 

Autonomous Management Cha nge Scores nn th, <; 
Ext ra Time Provided for Sm dg nLOuestions durin g Te a ^. lcd Revipws 
The teaching practice of allotting extra time for students to ask questions 
during the test review session is associated with increases in the relationship 
between Autonomous Management change scores and achievement (see Figure 8). 
This finding suggests that providing students with an opportunity to ask questions 
about the up-coming test enhanced the effectiveness of changes in Autonomous 
Management activities. 
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This result confirms our prediction that providing students with the 
opportunity to ask questions about the test can be a supportive teaching practice that 
informs students about the kinds of test items to expect and how to best study for 
the f sst. The research literature suggests that student expectations about task 
demands or learning goals influence their study activities and achievement (e.g. , 
Hamaker, 1986; Martin & Saljo., 1976; Rothkopf, 1973). 



Insert Figure 8 about here 



Memory Augmentation Change Scores o n the SAO: Number of Content Categories 

Covered on the Unit Test 
As the number of content categories on the test increases (see Figure 9), 
there is a corresponding decrease in the relationship between changes in Memory 
Augmentation scores and achievement test performance. The greater the number of 
categories on the test the less effective are changes in memory augmentation 
strategies. 

These results suggest that increasing the difficulty of the test by increasing 
the number of content categories covered may not lead to more productive memory 
augmentation strategies. A similar result was reported by other researchers 
(Entwistle & Ramsden, 1983; Entwistle & Tait, 1990) who found that students 
who described their courses as demanding in terms of workload tended to engage in 
more superficial, unproductive types of study strategies. In fact, Natriello ( 1983) 
suggested the relationship between workload and study practices is curvilinear. 
When the workload or requirements of the course become too demanding, students 
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resort to superficial types of study practices. This finding may be of considerable 
import for those who choose a test format comprised of many small items (e.g., 
multiple choice tests), which , by the nature of the item design task, tends to 
reinforce a view of the curriculum as merely an accumulation of finely distinguished 
objectives. 



Insert Figure 9 about here 



Discussion 

The findings suggest that an multi-level analysis provides a more 
comprehensive and revealing explanation of the hierarchical relationship between 
course characteristics, student study activities, and achievement. Had the pairs of 
variables been analyzed individually using traditional regression techniques, 
different patterns of results and implications of these findings would have emerged. 
For instance, there was a significant, positive < orrelation (.60) obtained between 
feedback on homework and achievement, suggesting that the more feedback 
provided on homework, the higher the achievement. However, the results of the 
multi-level analysis point to the diminishing returns of this type of feedback for 
promoting the effectiveness of particular study strategies. Thus, multi-level 
analyses paint a more accurate and sophisticated picture of how context variables 
impact the relationship between study activities and achievement. Ultimately, we 
are interested in arranging instructional environments that both promote productive 
study strategies and enhance achievement, and multi-level techniques allow 
researchers to examine all three iypes of variables simultaneously. 
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In terms of applications to the classroom, the results provide some 
potentially helpful suggestions concerning how biology educators might enhance 
the effectiveness of students' study behaviors. The following general summaries of 
the statistically significant findings would provide a good starting point for a 
confirmatory investigation. 

1. To increase the effectiveness of students' effort management activities (which 

would, one might presume, reinforce students' use of these effort 
management activities), teachers might provide enhanced feedback on pre- 
test quizzes, but refrain from doing so on homework. 

2. To increase the effectiveness of students' memory augmentation activities, 

teachers might refrain from preparing tests with large numbers of items that 
"cover" large numbers of concepts. 

3. To increase the effectiveness of students' autonomous management activities, 

teachers might provide time for students to ask questions during reviews, 
but, once again, refrain from providing enhanced feedback on homework. 
This summary of the results should not be seen as prescriptive in any way. 
First, the web of causality is not at all clear among the variables in the regressions. 
We have made causal assumptions that seem valid to us, but there is no safeguard 
in the present analysis against reasonable occurrences such as, teachers adapting 
their course characteristics to the particular sample of students in a particular class. 
Nor can we be sure that we have recorded all of the important course 
characteristics-for instance, we have relied heavily on information from the test and 
the period immediately preceding the test, which may be quite atypical, or quite 
inconsequential, in some classes. We must also note that, at the between-class 
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level, we have a very small sample size- 1 1 . Moreover, we conducted a large 
number of analyses, so we must be wary of ascribing too much importance to our 
findings. 

The most appropriate next step to take would be to replicate the study with a 
larger number of classes, seeing whether the results generalize to another sample, 
and, perhaps, determining the general izability of the findings to other biology units. 
Another area appropriate for further investigation is a more systematic and 
comprehensive study of the extent and content of the feedback provided to students 
on different academic tasks and its moderating effect on study activities and 
achievement. 
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Figure Captions 

Figure L A comparison of traditional research approaches and the multi-level 
approach to examine the relationships among course characteristics, study activities 
and achievement. 

Figure! Within-course regression slopes of GAT scores on engagement in Effort 
Management study activities (post-pre difference scores) as a function of feedback 
on quizzes. 

Figure 3. The fitted linear regressions relating changes in Effort Management to 
achievement for each value of Feedback on Quizzes for the five classrooms of 
Figure 2. 

Figure 4. The fitted linear regressions relating changes in Effort Management to 
achievement for each value of Feedback on Quizzes for all twelve classrooms. 
Figure 3, Within-course Coefficients of Regression of GAT scores on engagement 
in Effort Management study activities (post-pre difference scores) as a function of 
Feedback on Homework. 

Figure 6. The fitted linear regressions relating changes in Effort Management to 
achievement for each of the values for Feedback on Homework. 
Figure 7. Within course Coefficients of Regression of GAT scores on engagement 
in Autonomous Management study activities (post-pre difference scores) as a 
function of Feedback on Homework. 

Figure 8. Within-course Coefficients of Regression of GAT scores on engagement 
in Autonomous Management study activities (post-pre difference scores) as a 
function of Extra Time for Study Questions. 
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.Figure!. Within-course Coefficients of Regression of GAT scores on engagement 
in Memory Augmentation study activities (post-pre difference scores) as a function 
of Number of Content Categories on the Test. 
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